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Packet Processing Engine Architecture 



Background of the Invention 



16 1. Field of the Invention 



This invention relates to packet processing. 
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1 2. Related Art 
2 

3 In a computer network for transmitting information, messages are received 

4 by each router (or switch) at an input interface and retransmitted at an output interface, so 

5 as to forward those messages onward to their respective destinations. Each router per- 

6 forms a lookup operation for each message it encounters, in which the router determines 

7 from the message to which output interface the message should be forwarded. 

8 

p9 One problem in the known art is that the lookup operation can be relatively 

Ho complex, and can use a relatively large amount of processor resources. For example, the 
lookup operation can be complicated by concurrently determining one or more of the 

Qi2 following: 

Hi 3 

™14 • which output interface is the closest, within a defined network topology, to the 

~~ 

Q15 specified destination; 

16 

17 • whether the message is unicast or multicast, and in the latter case, from which in- 

18 put interface the message was received; 

19 

20 • whether the message is authorized to be forwarded by this router from its specified 

21 source, and whether the message is authorized to be forwarded by this router to its 

22 specified destination; 
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• whether the message should be forwarded to a selected output interface for quality 
of service considerations, security considerations, or other administrative consid- 
erations; 

• whether the message should be counted, measured, or otherwise accounted for, 
concurrently with forwarding. 

Known responses to this problem include (1) to provide greater processing 
capability, so as to make up for the processor load on the router, (2) to provide only some 
of these concurrent services, or to provide them only a reduced functionality. While these 
responses achieve the goal of routing messages in a forwarding network, they have the 
disadvantage that added services introduce additional load on the router processor and 
slow down the router. 

Accordingly, it would be advantageous to provide a method and system for 
packet processing that is not subject to drawbacks of the known art. 

Summary of the Invention 

The invention provides a method and system for packet processing, in 
which a router (or switch) is capable of processing incoming packets very quickly, thus 
performing level 2, 3, and 4 routing and switching, and substantial additional services, in 
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1 real time. A system includes a packet processing engine (PPE), having elements for re- 

2 ceiving packets, distinguishing header and payload information for those packets, 

3 outsourcing router decision-making to additional hardware resources (herein a "fast for- 

4 warding engine," or FFE), and ultimately forwarding those packets in response to out- 

5 sourced decisions. 

6 

7 In a first aspect of the invention, the PPE is time-synchronized to the FFE, 

8 so that the PPE can send and the FFE can receive packet routing information for decision- 
P 9 making at each one of a sequence of constant-duration time quanta. Similarly, the PPE 
'''-•ho can receive and the FFE can send packet routing information at each one of a sequence of 

1 similar time quanta. In addition to information about where to forward a packet, packet 
G12 routing information possibly also includes additional information such as packet treat- 
Mi 3 ment in response to access control, class of service or quality of service, accounting, and 
;J14 other administrative or managerial criteria. 

r"*s 

E5 15 

16 In a second aspect of the invention, the PPE and the FFE each have separate 

17 hardware resources allocated to their functions; these separate hardware resources can in- 

18 elude pin count, on chip memory, and transfer bandwidth to off-chip memory. This al- 

19 lows the PPE and the FFE to each perform their functions in parallel without substantial 

20 contention for operating resources. 

21 



Express Mailing EL524780450US 




102.1038.01 



1 In a third aspect of the invention, multiple PPE and FFE pairs can be incor- 

2 porated into a scaleable parallel system, operating in parallel to route (or switch) packets 

3 among a plurality of input and output interfaces. 

4 

5 In a preferred embodiment, the PPE includes separate treatment of packet 

6 header information and payload information, so the amount of information exchanged 

7 between the PPE and the FFE, and the amount of actual data movement performed by the 

8 PPE, can be relatively minimized. When determining the packet header information, the 
Q9 PPE can also parse the data packet (particularly what is conventionally called the packet 

header) and extract fields needed by the FFE to perform it's forwarding, ACL and QoS 

jSn decisions. In this way, the PPE reduces the amount of data that it needs to transmit to the 

-,ji2 FFE, thereby reducing the number of pins required by both the PPE and the FFE to im- 

Hi3 plement this communication. 
iii4 



pis Brief Description of the Drawings 

16 

17 Figure 1 shows a block diagram of a system for packet processing and 

18 packet forwarding. 

19 

20 Figure 2 shows a process flow diagram of a method of using a packet proc- 

21 essing element as in figure 1 . 

22 
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1 Figure 3 shows a block diagram of a system for parallel packet processing 

2 and packet forwarding. 

3 

4 Detailed Description of the Preferred Embodiment 

5 

6 In the following description, a preferred embodiment of the invention is de- 



7 scribed with regard to preferred process steps and data structures. Those skilled in the art 

8 would recognize after perusal of this application that embodiments of the invention can 
f « 9 be implemented using circuits adapted to particular process steps and data structures de- 
Sjio scribed herein, and that implementation of the process steps and data structures described 
^11 herein would not require undue experimentation or further invention. 

Gfl2 

U13 System Elements 
Q 14 



^15 Figure 1 shows a block diagram of a system for packet processing and 

16 packet forwarding. 

17 

18 A router 100 includes a set of input interfaces 1 1 1, a set of output interfaces 

19 1 12, a packet processing engine (PPE) 120, a PPE memory 130, and a fast forwarding en- 

20 gine (FFE) 140. The router 100 is coupled to one or more communication networks 160. 

21 
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The router 100 is disposed for routing (or switching) a sequence of packets 
170. Each packet 170 includes packet header information 171 and packet payload infor- 
mation 172. Each packet 170 ultimately has packet forwarding information 173 (not 
shown) decided for it, which is used for routing the packet 170. Each packet 170 might 
also have a packet index 174 (not shown) for reference purposes. 

Packet Processing Engine 

The PPE 120 is disposed to perform the following operations: 

• The PPE 1 20 receives input packets 1 70 at the input interfaces 111. 

The input interfaces 1 1 1 are coupled to at least one communication network 

160. 

• The PPE 120 distinguishes packet header information 171 from packet payload in- 
formation 172. 

In a preferred embodiment, input packets 170 and output packets 170 are 
modified using known packet modification protocols, for which there are known parsing 
rules. The PPE 120 uses these known parsing rules to distinguish packet header informa- 
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tion 171 from packet payload information 172. The PPE 120 extracts the packet header 
information and then stores that packet in the PPE memory 130. 

• The PPE 120 records packet header information 171 and packet payload informa- 
tion 172 in the PPE memory 130. 

In a preferred embodiment, the PPE 120 uses memory access bandwidth to 
reference the PPE memory 130 for recording and retrieving packet header information 
171 and. packet payload information 172 using the PPE memory 130. This allows the 
PPE 120 to refer to packets by a packet index 174. 

• The PPE 120 forwards packet header information 171 to the FFE 140. 

In a preferred embodiment, the PPE 120 is ready to forward packet header 
information 171 to the FFE 140 each two clock cycles. Each clock cycle is preferably 6-7 
nanoseconds. It may occur, for any individual incoming packet 170, that the PPE 120 
takes much longer than two clock cycles to distinguish packet header information 171 and 
packet payload information 172. However, the PPE 120 should have at least one new set 
of packet header information 171 for the FFE 140 at least that often. 

Similarly, in a preferred embodiment, the FFE 140 is ready to receive 
packet header information 171 from the PPE 120 each two clock cycles. It may occur, for 
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1 any individual incoming packet 170, that the FFE 140 takes much longer than two clock 

2 cycles to decide associated packet forwarding information 173. However, the FFE 140 

3 should be ready to receive one new set of packet header information 171 from the PPE 

4 1 40 at least that often. 

5 

6 • The PPE 120 receives packet forwarding information 173 for associated packet 

7 header information 171 from the FFE 140. 

8 

£39 In a preferred embodiment, the PPE 120 uses the packet index 174 to refer- 

^jio ence both packet header information 171 and associated packet payload information 172 

j=i 1 in the PPE memory 130. 

SJ12 

^713 • The PPE 120 modifies the packet to generate an output packet 170. 

pis In a preferred embodiment, the PPE 120 performs a rewrite operation on the 

16 packet 170. Rewrite operations include adjusting a TTL (time-to-live) IP field, , deter- 

17 mining a new CRC, rewriting the MAC-level addresses, and possibly other modifications 

18 of the fields.. Rewrite operations, and rewrite rules, are known in the art of Internet 

19 packet forwarding. 

20 

21 • The PPE 120 sends output packets 170 from the output interfaces 112. 

22 
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Similar to the input interfaces 111, the output interfaces 1 12 are also cou- 
pled to at least one communication network 160, preferably the same communication 
network 160 as the input interfaces 111. 

Fast Forwarding Engine 

The FFE 140 includes a packet information input port 141, a packet for- 
warding information output port 142, and is coupled to assistance devices for assisting in 
making packet forwarding decisions. 

The FFE 140 is coupled to a set of routing information memories 143 (in- 
cluding a spanning tree memory and a multicast expansion table), a forwarding content 
addressable memory (CAM) 144 and a forwarding memory 145, an input access CAM 
146 and an output access CAM 147, a CPU 148, and a net- flow routing engine 150. 

The FFE 140 is disposed to perform the following operations: 

• The FFE 140 receives packet header information 171 . 

• The FFE 140, with the assistance of the assistance devices, determines packet for- 
warding information 173 in response to packet header information 171. 
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1 In a preferred embodiment, the FFE 140 forwards the packet header infor- 

2 mation 171 to the forwarding CAM 144, which performs a lookup in its CAM entries to 

3 determine packet forwarding information 173 associated with the packet header informa- 

4 tion 171. Indices responsive to the lookup by the forwarding CAM 144 are recorded in 

5 the forwarding memory 145. 

6 

7 The FFE 140 accesses the forwarding CAM 144 to record new forwarding 

8 information rules as they become available, such as changes to the perceived network to- 
ri 9 pology, access control, and other administrative or managerial rules. The FFE 140 ac- 
Mio cesses the forwarding memory 145 to retrieve the packet forwarding information 173 as it 
1 yn is determined. 

Cj12 

1=^13 In a preferred embodiment, the forwarding CAM 144 includes a set of ter- 

^14 nary CAM entries. Each ternary CAM entry includes a set of bits which can match to 

^15 logical 0, to logical 1, or to either (that is, a "don't care" bit). Each ternary CAM entry is 

16 thus capable of being matched against the packet header information 171 to determine an 

17 index in the forwarding memory 145 of a set of packet forwarding information 173. . 

18 

19 In a preferred embodiment, this additional information is responsive to the 

20 IP source address, IP source address, IP source port, IP destination address, IP destination 

21 port, protocol type for the packet 1 70, and whether the packet 1 70 is unicast or multicast. 

22 
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In a preferred embodiment, the FFE 140 forwards an identifier for the input 
interface 1 1 1 at which the packet 170 was received to the input access CAM 146, to de- 
termine if access is permitted for the packet 170 at that input interface 111. 

Similarly, after determining an output interface for the packet 170, the FFE 
140 forwards an identifier for the output interface 112 at which the packet 170 was re- 
ceived to the input access CAM 146, to determine if access is permitted for the packet 
170 at that output interface 112. 

In a preferred embodiment, the packet forwarding information 173 includes 
how to forward the packet 170 (that is, to which output interface), as well as some or all 
of the following additional information: 

(1) what access control rules (that is, what ACL) to apply to the packet 170; 

(2) what class of service (CoS) and quality of service (QoS) rules to apply to 
the packet 170; 

(3) what accounting and statistics to keep regarding the packet 170 or the net 
flow that the packet 1 70 is part of; 
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(4) what other administrative or managerial rules or restrictions to apply to the 
packet 170. 

In a preferred embodiment, this additional information (and other additional 
services with regard to the packet 170) can be introduced without substantially adding to 
processing load on the FFE 140, as the forwarding CAM 144 and the forwarding memory 
145 provide pattern matching against the packet header information 171. 

• The network flow routing engine 150 provides network flow packet forwarding in- 
formation 173 to the FFE 140, if that network flow packet forwarding information 
173 is available. 

In a preferred embodiment, if the packet 170 can be routed using network 
flow information, the network- flow routing engine 150 independently determines net- 
workflow packet forwarding information 173 in response to the network flow associated 
with the packet header information 171 . If the network-flow routing engine 150 is able to 
determine that network flow packet forwarding information 173, the FFE 140 uses the 
network flow packet forwarding information 1 73 in place of packet forwarding informa- 
tion 173 it might otherwise determine for itself. 
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1 Method of Operation 
2 

3 Figure 2 shows a block diagram of a packet processing element in a system 

4 as in figure 1 . 

5 

6 A method 200 includes a set of flow points and a set of steps. The system 



7 100 performs the method 200. Although the method 200 is described serially, the steps of 

8 the method 200 can be performed by separate elements in conjunction or in parallel, 
_* n 9 whether asynchronously, in a pipelined manner, or otherwise. There is no particular re- 

fear 

H|io quirement that the method 200 be performed in the same order in which this description 

^11 lists the steps, except where so indicated. 

sits 

H=13 At a flow point 210, the PPE 120 is ready to receive input packets 170 at 

: ; ; 

m 14 the input interfaces 111. 

1 15 

16 At a step 21 1, the PPE 120 receives an input packet 170 at one of the input 

17 interfaces 111. 

18 

19 At a step 212, the PPE 120 parses the packet 170 to distinguish a packet 

20 header from a remainder of the packet, and to determine those portions of the packet 

21 header that are relevant to packet routing. This allows the PPE 120 to distinguish packet 

22 header information 171 from packet payload information 172. The packet 170 is not af- 
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1 fected by this parsing. The entire packet 170 remains stored in the PPE memory 130 as 

2 one unit. 

3 

4 For example, in a preferred embodiment, the PPE 120 determines the IP 

5 source address, IP source port, IP destination address, IP destination port, protocol type 

6 for the packet 170, and whether the packet 170 is unicast or multicast. In a preferred em- 

7 bodiment, these values are treated as packet header information 171. 

8 

n 9 At a step 213, the PPE 120 forwards packet header information 171 for the 

Siio packet 170 to the FFE 140. As part of this step, the FFE 140 receives packet header in- 

LLf 

rUi 1 formation 1 7 1 for the packet 1 70 from the PPE 1 20. 

Si 12 

M>13 At a step 214, the FFE 140 sends packet forwarding information 173 for the 

Hi4 packet 170 to the PPE 120. As part of this step, the PPE 120 receives packet forwarding 

JSi5 information 173 for the packet 170 from the FFE 140. 

16 

17 At a step 215, the PPE 120 associates the packet forwarding information 

18 173 received from the FFE 140 with the packet 170, using the packet index 174. 

19 

20 At a step 216, the PPE 120 rewrites the packet 170 using the packet for- 

21 warding information 173 and a set of rewrite rules for the packet 170. As noted above, 
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1 rewrite operations include adjusting a hop count for the packet, determining a new CRC, 

2 and possibly other protocol reformatting operations. 

3 

4 At a step 217, the PPE 120 sends the packets 170 to the output interface 1 12 

5 indicated by the packet forwarding information 173. 

6 

7 After a flow point 218, the PPE 120 has sent the packet 170 to a designated 

8 output interface 112. 



□ 9 

Mio Parallel System 
§ ; «11 



~ ji2 Figure 3 shows a block diagram of a system for parallel packet processing 

Mi 3 and packet forwarding. 

S 14 

?515 A system 300 for parallel packet processing and packet forwarding includes 

16 a plurality of interfaces 1 10, a plurality of routing pairs 320, and a cross-bar switch 330. 

17 

18 Each plurality of interfaces 1 10 includes a set of input interfaces 1 1 1 and a 

19 set of output interfaces 112. Packets 170 can be received at the input interfaces 1 1 1 and 

20 can be sent using the output interfaces 112. 

21 
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Each routing pair 320 includes a matched PPE 120 and FFE 140, and asso- 
ciated memories and assistance devices, as described with reference to figure 1 . 



The cross-bar switch 330 is coupled to outputs from each PPE 120 in each 
matched routing pair 320. 

When a packet 170 is received at a particular interface 110 (and thus at a 
particular input interface 1 1 1 therein), they are coupled to the routing pair 320 associated 
with that particular interface 1 10. 

When a packet 170 is received at a particular routing pair 320, it is received 
by the PPE 120 in that particular matched routing pair 320. The PPE 120 and the FFE 
140 in that particular routing pair 320 cooperate to route (or switch) and otherwise proc- 
ess the packet 170 as described with regard to figure 1 and figure 2. 

When a packet 170 is output from a routing pair 320, the PPE 120 forwards 
the packet 170 to the crossbar switch 330 with instructions indicating a particular desti- 
nation interface 110. The crossbar switch 330 provides flow control between different 
routing pairs 320 so that multiple routing pairs 320 do not simultaneously send packets 
170 to the same output interface 112 and overrun buffering therein. 



Express Mailing EL524780450US 



17 



102.1038.01 

When a packet 170 arrives at the cross-bar switch 330, the cross-bar switch 
330 forwards that packet 170 to its destination interface 1 10 5 at which it is output from its 
destination output interface 112. 

Alternative Embodiments 

Although preferred embodiments are disclosed herein, many variations are 
possible which remain within the concept, scope, and spirit of the invention, and these 
variations would become clear to those skilled in the art after perusal of this application. 
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